Multi-dimensional model reconstruction

ABSTRACT

Systems and methods are disclosed for adjusting plane positions in multi-dimensional models. Disclosed is moving a plane associated with an architectural element based on a scale and a translation positional error, wherein the scaled is determined based on the architectural element, and the translation position error is based on a position of the architectural element, and reconstructing the multi-dimensional building model based on the moved plane.

CROSS REFERENCE TO RELATED PATENTS/PATENT APPLICATIONS

The present U.S. Utility Patent application claims priority pursuant to35 U.S.C. § 120 as a continuation of U.S. Utility application Ser. No.16/578,964, entitled “ADJUSTMENT OF ARCHITECTURAL ELEMENTS RELATIVE TOFACADES,” filed Sep. 23, 2019, which is a continuation of U.S. Utilityapplication Ser. No. 16/538,386, entitled “ADJUSTMENT OF ARCHITECTURALELEMENTS RELATIVE TO FACADES,” filed Aug. 12, 2019, issued as U.S. Pat.No. 10,515,434 on Dec. 24, 2019, which is a continuation of U.S. Utilityapplication Ser. No. 15/817,620, entitled “MULTI-DIMENSIONAL MODELDIMENSIONING AND SCALE ERROR CORRECTION,” filed Nov. 20, 2017, issued asU.S. Pat. No. 10,475,156 on Nov. 12, 2019, which is a continuation ofU.S. Utility application Ser. No. 15/400,718, entitled“MULTI-DIMENSIONAL MODEL DIMENSIONING AND SCALE ERROR CORRECTION,” filedJan. 6, 2017, issued as U.S. Pat. No. 9,830,681 on Nov. 28, 2017, whichis a continuation-in-part of U.S. Utility application Ser. No.15/332,481, entitled “SCALE ERROR CORRECTION IN A MULTI-DIMENSIONALMODEL,” filed Oct. 24, 2016, issued as U.S. Pat. No. 9,830,742 on Nov.28, 2017, which is a continuation of U.S. Utility application Ser. No.14/610,850, entitled “SCALE ERROR CORRECTION IN A MULTI-DIMENSIONALMODEL,” filed Jan. 30, 2015, issued as U.S. Pat. No. 9,478,031 on Oct.25, 2016, which claims priority pursuant to 35 U.S.C. § 119(e) to U.S.Provisional Application No. 61/934,541, entitled “SCALE ERROR CORRECTIONIN A GEO-REFERENCED THREE-DIMENSIONAL (3D) MODEL,” filed Jan. 31, 2014,all of which are hereby incorporated herein by reference in theirentirety and made part of the present U.S. Utility Patent Applicationfor all purposes.

This application makes reference to the complete subject matter of U.S.Utility patent application Ser. No. 13/624,816 entitled“THREE-DIMENSIONAL MAP SYSTEM” filed Sep. 21, 2012, issued as U.S. Pat.No. 8,878,865 on Nov. 4, 2014, and U.S. patent application Ser. No.12/265,656 entitled “METHOD AND SYSTEM FOR GEOMETRY EXTRACTION, 3DVISUALIZATION AND ANALYSIS USING ARBITRARY OBLIQUE IMAGERY” filed Nov.5, 2008, issued as U.S. Pat. No. 8,422,825 on Apr. 16, 2013, both ofwhich are incorporated herein by reference in their entirety.

BACKGROUND Technical Field

The technology described herein relates generally to a system and methodfor constructing or reconstructing a multi-dimensional model usingpositions of architectural elements relative to facades based on knownarchitectural standards.

Description of Related Art

Location-based technologies and mobile technologies are often consideredthe center of the technology revolution of this century. Essential tothese technologies is a way to best present location-based informationto devices, particularly mobile devices. The technology used torepresent this information has traditionally been based on atwo-dimensional (2D) map. Some efforts have been made to generatethree-dimensional (3D) maps of urban cities with accurate 3D texturedmodels of the buildings via aerial imagery or specializedcamera-equipped vehicles. However, these 3D maps have limited textureresolution, geometry quality, inaccurate scaling and are expensive, timeconsuming and difficult to update and provide no robust real-time imagedata analytics for various consumer and commercial use cases.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates one embodiment of a system architecture in accordancewith the present disclosure;

FIG. 2 illustrates a flowchart representing one embodiment of a processfor accurately rescaling a multi-dimensional building model inaccordance with the present disclosure;

FIG. 3 illustrates a flowchart representing another embodiment of aprocess for accurately scaling/rescaling a multi-dimensional buildingmodel in accordance with the present disclosure;

FIG. 4 illustrates a flowchart representing one embodiment of a processfor accurately scaling/rescaling/repositioning one or more planes of amulti-dimensional building model in accordance with the presentdisclosure;

FIG. 5 illustrates an example embodiment for identifying scale/scaleerror in a multi-dimensional building model using siding rows as theknown architectural dimension in accordance with the present disclosure;

FIG. 6 illustrates an example embodiment for identifying scale/scaleerror in a multi-dimensional building model using a door as the knownarchitectural dimension in accordance with the present disclosure;

FIG. 7 illustrates yet another example embodiment for identifyingscale/scale error in a multi-dimensional building model using bricklayout as the known architectural dimension in accordance with thepresent disclosure;

FIG. 8A-8D illustrate additional example embodiments for identifyingscale/scale error in a multi-dimensional building model using roofingelements as the known architectural dimension in accordance with thepresent disclosure;

FIG. 9 illustrates yet another example embodiment for identifying scaleerror in a multi-dimensional building model using known positions ofelements as the known architectural dimension in accordance with thepresent disclosure;

FIG. 10 illustrates an embodiment of a flowchart for improving theaccuracy of the dimensions of a building model in accordance with thepresent disclosure;

FIG. 11 illustrates an embodiment of a flowchart for weighing variousknown architectural elements to scale dimensions and/or adjust scaledimensions of a multi-dimensional building model in accordance with thepresent disclosure; and

FIG. 12 illustrates a diagrammatic representation of a machine in theexample form of a computer system in accordance with the presentdisclosure.

DETAILED DESCRIPTION

FIG. 1 illustrates one embodiment of system architecture in accordancewith the present disclosure. In one embodiment, image processing system100 includes image processing servers 102. Image database (DB) 104 andimage processing servers 102 are coupled via a network channel 106.

The network channel 106 is a system for communication. Network channel106 includes, for example, an Ethernet or other wire-based network or awireless NIC (WNIC) or wireless adapter for communicating with awireless network, such as a WI-FI network. In other embodiments, thenetwork channel 106 includes any suitable network for any suitablecommunication interface. As an example, and not by way of limitation,the network channel 106 can include an ad hoc network, a personal areanetwork (PAN), a local area network (LAN), a wide area network (WAN), ametropolitan area network (MAN), or one or more portions of the Internetor a combination of two or more of these. One or more portions of one ormore of these networks may be wired or wireless. As another example, thenetwork channel 106 can be a wireless PAN (WPAN) (such as, for example,a BLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a 3G or 4Gnetwork, a cellular telephone network (such as, for example, a GlobalSystem for Mobile Communications (GSM) network).

In one embodiment, the network channel 106 uses standard communicationstechnologies and/or protocols. Thus, the network channel 106 can includelinks using technologies such as Ethernet, 802.11, worldwideinteroperability for microwave access (WiMAX), 3G, 4G, CDMA, digitalsubscriber line (DSL), etc. Similarly, the networking protocols used onthe network channel 106 can include multiprotocol label switching(MPLS), the transmission control protocol/Internet protocol (TCP/IP),the User Datagram Protocol (UDP), the hypertext transport protocol(HTTP), the simple mail transfer protocol (SMTP), and the file transferprotocol (FTP). In one embodiment, the data exchanged over the networkchannel 106 is represented using technologies and/or formats includingthe hypertext markup language (HTML) and the extensible markup language(XML). In addition, all or some of links can be encrypted usingconventional encryption technologies such as secure sockets layer (SSL),transport layer security (TLS), and Internet Protocol security (IPsec).

In one or more embodiments, the image processing servers 102 includesuitable hardware/software in the form of circuitry, logic gates, and/orcode functions to process digital images to include, but not limited to,calculation of one or more image measurements according to anarchitectural feature measurement located within the images themselves.Capture device(s) 108 is in communication with image processing servers102 for collecting digital images of building objects. Capture devices108 are defined as digital devices for capturing images. For example,the capture devices include, but are not limited to: a camera, a phone,a smartphone, a tablet, a video camera, a security camera, aclosed-circuit television camera, a computer, a laptop, a webcam,wearable camera devices, photosensitive sensors, drone mounted imagingdevices, equivalents or any combination thereof.

Image processing system 100 also provides for viewer device 110 that isdefined as a display device. For example, viewer device 110 can be acomputer with a monitor, a laptop, a smartphone, a tablet, a touchscreen display, an LED array, a television set, a projector display, awearable heads-up display of some sort, a remote display associate witha camera device, or any combination thereof. In one or more embodiments,the viewer device includes display of one or more building facades andassociated measurements, such as, for example, a mobile device, aconventional desktop personal computer having input devices such as amouse, keyboard, joystick, or other such input devices enabling theinput of data and interaction with the displayed images and associatedmeasurements.

In one embodiment, ground-level images of a physical building areuploaded to image processing system 100 from a capture device. Anuploaded image is, for example, a digital photograph of a physicalbuilding showing a façade (side) of the physical building. Imageprocessing system 100 is used to generate accurately textured, 2D/3Dbuilding models based on the collected digital images. The textured,2D/3D building models are generated using systems and methods, forexample, as provided in U.S. Pat. Nos. 8,878,865, and 8,422,825, andhereby incorporated by reference. In addition, third party sources oftextured models can be substituted in the various embodiments describedherein without departing from the scope of the technology described.

However, orthogonal/oblique imagery typically used for geo-referencingis inherently inaccurate and/or of low resolution resulting in scaled2D/3D models that are not accurate enough to extrapolate precisedimensions. For example, a scaled 3D model can be used to calculatedimensions for building materials (i.e., siding for an exterior wall,exterior brick, a door, etc.) in a construction project. Using anorthogonally geo-referenced 3D model, the calculated dimensions arelikely to include error given the low resolution and potential forvisibility errors (e.g., occlusions).

It is known that some architectural elements used in the constructionindustry are standardized. For example, rows of siding applied toexterior walls are typically separated by 6-10 inches (depending on thetype of siding and location). However, it is understood that otherdimensions exist and that the technology described herein is not limitedto the specific dimensions provided in example embodiments.

In one or more embodiments of the technology described herein, a systemand method are provided for dimensioning and/or correcting error in anuntextured/textured multi-dimensional building model. Images of amulti-dimensional building model are used to identify scale and/or scaleerror by comparing to known architectural dimensions. Once scale orscale error is identified, the textured models are reconstructed withaccurately scaled multi-dimensional building models. Throughout thedescriptions and example embodiments that follow, a measurement of oneor more known architectural elements can be used to scale one or moreimage planes of a multi-dimensional model while it is being constructedor after construction is completed. In addition, scale errors can bedetermined based on one or more known architectural elements and used torescale one or more image planes of the multi-dimensional model.

FIG. 2 illustrates a flowchart representing one embodiment process foraccurately scaling/rescaling a textured multi-dimensional building modelin accordance with the present disclosure. In step 201, at least onedigital image is retrieved (e.g., image associated with buildingobject). In one embodiment, only a portion of the digital image isretrieved since the entire image (façade) may not be needed forscaling/scale error corrections. This portion, for example front façade,may include a cut-out of a full 2D image that has been rectified andcorrelated to vertices of geometric planes/polygons that make up aportion of a 3D model. For example, the portion may be a close-up of thefront porch of a house that includes the front door. In step 202, knownarchitectural elements of the digital image are identified. In oneembodiment, architectural elements are identified using known image orobject recognition techniques, including those techniques of the USreferences incorporated herein by reference. In alternative embodiments,the identification of architectural elements is accomplished using otherapproaches. For example, the boundaries for rows of siding areautomatically identified using line detection techniques (e.g.,frequency domain filtering). In step 203, boundaries (i.e., siding,bricks, etc.) are identified (defined) using unique feature detectionmethods that look for repeated patterns, such as, consistent parallellines or line intersections. For yet another example, boundaries forarchitectural elements are detected using unsupervised clusteringalgorithms based on learned, predictable patterns. For yet anotherexample, boundaries can be manually marked up (e.g., by human observer).

A measurement (e.g., dimensional ratios) of the architectural element isconducted in step 204 using image processing system 100 of FIG. 1 . Inone embodiment, siding rows are used as the known architectural elementand the distance between siding rows (top boundary and bottom boundary)is measured in the multi-dimensional model. Pixels defining theboundaries of the architectural elements are identified within themulti-dimensional building model. In one embodiment, a plurality ofmeasurements is conducted to determine an average measurement betweenpixels representing the boundaries of a siding row. In one embodiment, aplurality of measurements is conducted to determine an averagemeasurement between pixels representing the boundaries of a siding row.

The calculated average measurement value of the known architecturalelement is compared to a threshold measurement (measurementincluding+−threshold error) according to known architectural standarddimensions in step 205. The threshold measurement accounts for theinherent inaccuracy of the imagery (scale error) and provides a likelyrange of values that are used to correlate the average measurement valueto an actual measurement value (real dimensions based on knownarchitectural standard dimensions). For example, if the knownarchitectural standard dimensions for a solar panel is 10×10 (feet), thethreshold will be established using, for example, +/−10% of the 10 feet(or up to 1 ft) in both directions (x and y). If the average measurementfalls within the threshold measurement, it is assumed that the averagemeasurement is likely to be the known architectural standard dimension.If the average measurement fails to fall within the thresholdmeasurement, it is assumed that it does not apply to the knownarchitectural standard or it is from a different standard dimension(i.e., 5×5, 15×15, etc.).

In one embodiment, the known architectural standard dimension is adistance between rows of siding on the façade of a building. Aspreviously discussed, the boundaries (top and bottom edges) for rows ofsiding applied to exterior walls of a building object are frequentlyseparated by 6-10 inches (depending on the type of siding). In oneembodiment, a digital image having siding exposed on at least oneexterior wall is provided that corresponds to a textured façade of amulti-dimensional building model. Image processing system 100 of FIG. 1identifies the exterior siding using known computer vision techniques,defines the boundaries of the siding rows, correlates the pixels to thedefined boundaries and measures the distance between the boundarydefining pixels of adjacent rows. In one embodiment, the distancebetween the boundaries is measured in a plurality of locations on theexterior wall to create an average measurement value. In one or moreembodiments, dimensional ratios are used to determine proper scaling.

The average measurement value of the boundaries between the rows ofsiding is compared to known architectural standard dimensions of, forexample, 6 inches, 7 inches, 8 inches, etc. separating each row from anadjacent row. For example, an average measurement of 6.63 inchesindicates ambiguity whether it is actually 6 inches (would representapproximately 10.5% error) or 7 inches (would represent approximately5.3% error) as the architectural dimension standards indicate. In oneembodiment, the average measurement falls within a given threshold rangeof +/−I 0% (inherent orthogonal imagery error).

As previously discussed, an average measurement value of 6.63 inches isindicative that the siding may represent either the 6 inch or 7 incharchitectural standard dimension. To determine actual dimensions, theaverage measurement value is compared to the threshold ranges for both 6inches and 7 inches. In order for the 6.63 inch average measurement tobe correlated to an actual measurement of 6 inches, the averagemeasurement would have to fall between 5.4 inches and 6.6 inches (0.6inches=10%). The average measurement of 6.63 (i.e., 10.5%) is outside ofthe threshold and, therefore, the rows of siding are not correlated to a6 inch distance between rows. While described using a +/−10% threshold,other thresholds are envisioned without departing from the scope of thetechnology described herein.

In the same example, in order for the average measurement to becorrelated to an actual measurement of 7 inches, the average measurementwould have to fall between the threshold range of 6.3 inches and 7.7inches. The average measurement of 6.63 inches (i.e., 5.3%) fallsbetween the threshold so it is determined that the distance between rowsof siding has high likelihood of an actual value of 7 inches.

In another embodiment, a door is used as the known architecturalstandard dimension. There are various sizes of doors used in theconstruction industry (e.g., single doors, French doors, etc.). In oneexample, a typical door size may be 30×80 (i.e., 30 inches wide by 80inches high). It is understood by those skilled in the art that thetechnology described here includes, but is not limited to, commonly useddoor sizes (e.g., the most common widths are 28, 30 and 32 inches;typically, around 80 inches.)

Using the process described above, a corresponding digital image is usedto identify the door as an architectural element on at least one façadeof the building model. In one embodiment, an average measurement of thewidth and the height is determined and compared to the threshold. In oneembodiment, a width-to-height ratio or height-to-width ratio of thearchitectural element is determined and compared to a known ratio,including error threshold. In another embodiment, the total area of thedoor is determined and used as the average measurement value. Using thecomparison of the average measurement to the measurement with thresholderror, the actual door size is determined based on the known doorstandard dimensions.

In yet another embodiment, bricks sizes are used as the knownarchitectural standard dimensions. There are various sizes of bricksused in the construction industry (e.g., standard, modular, Norman,Roman, jumbo, etc.). The size of the brick is used to extrapolate a wallsize and identify error in a multi-dimensioned building model. Forexample, a typical brick dimension is 3½×2¼×8 (depth (D)×height(H)×length (L) in inches). However, it is understood by those skilled inthe art that the technology described here includes, but is not limitedto, other commonly used bricks dimensions listed in Table 1.

TABLE 1 brick types and standard dimensions. Actual Size Brick Type D ×H × L (inches) Modular 3½ × 2¼ × 7½ Norman 3½ × 2¼ × 11½ Roman 3½ × 1¼ ×11½ Jumbo 3½ × 2½ × 8 Economy 3½ × 3½ × 7½ Engineer 3½ × 2¾ × 7½ King 3× 2¾ × 9¾ Queen 3 × 2¾ × 8 Utility 3½ × 3½ × 11½

In a similar process to the previously discussed embodiment of usingsiding rows as the known architectural standard dimension, brick heightand width is used to identify error in the building model. An averagemeasurement of a distance between rows of bricks (from the bottom of onebrick to the bottom of the brick above including or, optionally,excluding mortar) is compared to known architectural standard dimensionsseparating each row from a subsequent row. An average measurement valueof the multi-dimensional building model's brick façade is determined andcompared to the measurements, including threshold error values, forknown architectural dimensions separating the brick rows. Thresholdvalues are established for each of the brick types and the comparison ismade between the average measurement value and the known architecturalstandard dimension+−threshold error. In other embodiments, a brick'swidth or width and height, width-to-height, or height-to-width ratio iscompared against known dimensional architectural standards (with orwithout error thresholds).

In step 206, the determined actual dimension of a known architecturalelement is used to scale/rescale and reconstruct the multi-dimensional(2D/3D) building model. In one embodiment, an untextured, building modelis scaled/rescaled using one of the vertices as an anchor point. Forexample, once a known architectural measurement is determined (e.g.,based on 30:80 (3:8) ratio of front door), that scale (30 inches forwidth of door and 80 inches for height of door) is used for acorresponding door image measurement and thereafter is used to scale oneor more image planes of the multi-dimensional model. In a rescalingexample, if the actual dimension determines that building model contains10% error (too large), a vertex is used as an anchor point and thelength of one of the lines/edges corresponding to the vertex is reducedby 10%. Once the edge has an accurate dimension, the vertex is anchored(i.e., anchored in a real-world position). The dimensions and positionof the remaining vertices and edges are adjusted accordingly to maintainthe original geometry (angles of the vertices) of the building model. Inanother embodiment, a centroid (the geometric center of the buildingmodel) is used as an anchor point and the dimensions of the vertices andedges are adjusted accordingly. Once a scaled/rescaled building modelhas been constructed/reconstructed, the building model is textured basedon the digital images with the original digital images used fortextures.

FIG. 3 illustrates a flowchart representing another embodiment of aprocess for accurately scaling/rescaling a multi-dimensional buildingmodel in accordance with the present disclosure. For example, a randomobject in a field-of-view (e.g., foreground/background) during imagingof a subject building object is used to properly dimension (scale) oneor more planes within a multi-dimensional model of the building object.

In step 301, at least one digital image is retrieved (e.g., imageassociated with building object). For example, the digital image may bean image of a house that includes the front yard surrounding a frontfaçade of the house that is the subject of the multi-dimensionalbuilding model.

In step 302, foreground/background objects in a field of view of thebuilding object are identified. In one embodiment, foreground/backgroundobjects are identified using known image or object recognitiontechniques, including those techniques of the US references incorporatedherein by reference. In alternative embodiments, the identification offoreground/background objects is accomplished using other approaches.For example, parked automobiles, light fixtures, lamp posts, telephonepoles, stop signs, play structures, tables, chairs, or lawn equipmentare automatically identified using image recognition techniques. In step303, boundaries of these foreground/background objects are identifiedusing unique feature detection methods that look for repeated patterns,such as, consistent parallel lines or line intersections. For yetanother example, boundaries for foreground/background objects aredetected using unsupervised clustering algorithms based on learned,predictable patterns. For yet another example, boundaries can bemanually marked up (e.g., by human observer).

A measurement (e.g., dimensional ratio) of the foreground/backgroundobject is conducted in step 304 using image processing system 100 ofFIG. 1 . In one example embodiment, telephone poles within the digitalimage are used as the known foreground/background object and the knowndimensional ratio (e.g., ratio of height-to-width) of a standardtelephone pole is used to scale/rescale (step 306) one or more imageplanes of the associated multi-dimensional building model. In oneembodiment, a plurality of telephone pole ratios located within theimage is averaged to determine an average ratio of a plurality oftelephone poles. In addition, a distance from the pole to a plane of thebuilding (house) is used to geometrically calculate a relativemeasurement relationship using known geometric image processing methods.

The calculated average ratio value of the known foreground/backgroundobjects is compared to a threshold ratio (ratio with threshold error)according to known standard dimensions in step 305. The thresholdmeasurement accounts for the inherent inaccuracy of the imagery (scaleerror) and provides a likely range of values that are used to correlatethe average ratio value to an actual ratio value (real dimensions basedon known standard dimensions). For example, if the known standard heightfor a class 6 telephone pole in the United States is about 40 ft. (12 m)long and it is buried about 6 ft. (2 m) in the ground, 34 ft. will beused for a standard height, the threshold will be established using, forexample, +/−10% of the 34 feet (or up to 3.4 ft.). The minimum width fora class 6, 40 foot pole is 9.07 inches measured at six feet from thebutt end. Therefore, the width threshold will be established using, forexample, +/−10% of the 9.07 inches (or up to 0.907 inches). If theaverage ratio falls within the threshold, it is assumed that the averagemeasurement is likely to be the known standard dimension. If the averagemeasurement fails to fall within the threshold, it is assumed that itdoes not apply to the known standard or it is from a different standarddimension (i.e., 50 ft, 60 ft, class 7, class 8, etc.).

In one embodiment, a width-to-height ratio or height-to-width ratio ofthe foreground/background object is determined and compared to a knownthreshold ratio. In another embodiment, the total area of the object isdetermined and used as the average measurement value. Using thecomparison of the average measurement to the threshold measurement, theactual object is determined based on the known standard dimensions.

In step 306, the determined actual dimension of a known architecturalelement is used to scale/rescale and reconstruct the multi-dimensional(2D/3D) building model. In one embodiment, an untextured, building modelis scaled/rescaled using one of the vertices as an anchor point. Forexample, once a known measurement is determined, that scale (e.g., 34feet) is used for a corresponding telephone pole image measurement andthereafter is used to scale one or more image planes of themulti-dimensional model (as corrected based on distance from plane). Ina rescaling example, if the actual dimension determines that themulti-dimensional building model contains 10% error (too large), avertex is used as an anchor point and the length of one of thelines/edges corresponding to the vertex is reduced by 10%. Once the edgehas an accurate dimension, the vertex is anchored (i.e., anchored in areal-world position). The dimensions and position of the remainingvertices and edges are adjusted accordingly to maintain the originalgeometry (angles of the vertices) of the building model. In anotherembodiment, a centroid (the geometric center of the building model) isused as an anchor point and the dimensions of the vertices and edges areadjusted accordingly. Once a scaled/rescaled building model has beenconstructed/reconstructed, the building model is textured based on thedigital images with the original digital images used for textures.

FIG. 4 illustrates a flowchart representing one embodiment of a processfor accurately scaling/rescaling/repositioning one or more planes of amulti-dimensional building model in accordance with the presentdisclosure. Sometimes, errors in positioning of various image planeswithin a multi-dimensional model occur due to skew, obfuscation (i.e.,hidden planes), or missing images of the subject plane. However, if keyknown architectural features within the multi-dimensional model can beidentified, the model can be corrected by properly scaling/rescaling ormoving selected planes (e.g., walls) to correct positions based on knownrelationships between the key known architectural features andassociated planes in the multi-dimensional building model. For example,downspouts always follow an exterior wall edge. If the downspouts can beproperly identified, proper placement of associated wall planes can beimproved. In another example embodiment, gables are symmetrical andtherefore walls supporting the gables should also be positionedsymmetrically (e.g., equidistant from the center of the gable).Therefore, if one half of a symmetrical architectural feature isproperly identified and dimensioned, for example, one side of a gable,everything is known to draw the other side of the gable. Once known, thedimensions of the gable can be used to properly size other architecturalfeatures, for example, a garage door under the gable.

In step 401, at least one digital image is retrieved (e.g., imageassociated with building object). For example, the digital image may bean image of a front building façade. In one embodiment, only a portionof the digital image is retrieved since the entire image (façade) maynot be needed for scaling/scale error/plane positioning corrections.This portion, for example front façade, may include a cut-out of a full2D image that has been rectified and correlated to vertices of geometricplanes/polygons that make up a portion of a 3D model. For example, theportion may be a close-up of the outside wall edge of a house thatincludes a downspout.

In step 402, key known architectural elements are identified. In oneembodiment, architectural elements are identified using known image orobject recognition techniques, including those techniques of the USreferences incorporated herein by reference. In alternative embodiments,the identification of architectural elements is accomplished using otherapproaches. For example, downspouts are automatically identified usingline detection techniques (e.g., frequency domain filtering) or usingunique feature detection methods that look for repeated patterns, suchas, consistent parallel lines or line intersections. For yet anotherexample, key known architectural elements are detected usingunsupervised clustering algorithms based on learned, predictablepatterns. For yet another example, key known architectural elements canbe manually marked up (e.g., by human observer).

An identification of the position (included boundaries) of the key knownarchitectural element is conducted in step 403 using image processingsystem 100 of FIG. 1 . In one embodiment (see FIG. 9 and associateddiscussion), downspouts are used as the key known architectural element.

In step 404, the position of the known architectural element (e.g., atleast an inside edge (nearest exterior of wall plane)) is compared to aposition of an exterior wall plane associated with the downspout. Thecomparison accounts for the inherent inaccuracy of the imagery andprovides a likely position of the exterior wall plane (based on a knownrelationship between the downspout and the exterior wall plane),juxtaposed with the downspout (adjusting for a predetermined/calculatedknown gap between downspout and wall).

In step 405, the determined actual positioning of the key knownarchitectural element is used to scale/rescale/reposition image planesand reconstruct the multi-dimensional (2D/3D) building model. Once ascaled/rescaled/repositioned plane building model has beenconstructed/reconstructed, the building model is textured based on thedigital images with the original digital images used for textures.

FIG. 5 illustrates an example embodiment for identifying error in abuilding model using siding rows as the known architectural dimension inaccordance with the present disclosure. Digital image 501 captures abuilding object showing a façade covered with siding. Using thetechniques described previously, corresponding to the boundaries of eachrow of siding are determined. In one embodiment, the boundaries of theentire row (e.g., left edge of the façade to right edge of the façade)of siding are determined. In other embodiments, less than an entire rowis used. Lines 502 and 503 define the upper and lower boundary of onerow of siding. Measurement 504 determines the distance between thepixels defining the upper and lower boundary of the row of siding. Inone embodiment, the number of pixels between the upper and lowerboundary of the row of siding is determined and, based on the resolutionof each pixel, a measurement (e.g., feet (f), inches (in), meters (m),centimeters (cm), etc.) can be extrapolated. In another embodiment, theboundaries of a portion of the siding row are determined. Lines 505 and506 define the upper and lower boundaries of a portion of the row ofsiding. Measurement 507 determines the distance between the pixelsdefining the upper and lower boundaries of the row of siding. In yetanother embodiment, measurements 504 and 507 are averaged to determinean average measurement of the distance between rows of siding.

In one embodiment, the camera orientation relative to the facades of thebuilding in the image is solved using known methods. The building facadeorientation information is used to skew (i.e., rectify) the digitalimage to appear as if it was captured from directly in front of thecorresponding façade of the building. By solving for the building facadeorientations and rectifying the digital image, the siding rows areparallel lines and do not converge to cause error in the averagemeasurements across the length of the façade. In another embodiment, thedigital image is a rectified digital image created during modelconstruction and texturing.

FIG. 6 illustrates an example embodiment for identifying error in abuilding model using a door as the known architectural dimension inaccordance with the present disclosure. Digital image 601 shows a frontfaçade of a building object that includes a door. The boundaries of theknown architectural feature (i.e., the door) are determined and adimension (dimensional ratio) is extrapolated based on door 602boundaries. The ratio between the door height and width is used toidentify error and determine the actual dimensions of the door. As withthe previous examples, the measurements are compared to existing knowndoor dimensions and/or ratios, including threshold error rates, toidentify a possible known architectural standards match. FIG. 6 is usedfor diagrammatic purposes, specific positioning and designation ofdimensions may change from image-to-image and distinct architecturalfeature. For example, the known door architectural standard may includeor not include the door frame as shown. For example, when an externalstorm door is also included, in some embodiments, measurements will betaken of the storm door only, the exterior door or both.

FIG. 7 illustrates yet another example embodiment for identifying errorin a building model using a brick layout as the known architecturaldimension in accordance with the present disclosure. Digital image 701captures a façade of a building object showing exposed brick. In oneembodiment, a portion of the façade is used to establish the top andbottom boundaries of a horizontal row of brick in a similar technique tothe previously discussed siding rows. Portion 702 shows an exposed brickfaçade with bricks in the traditional offset pattern. Top boundary 703and bottom boundary 704 are determined for a row of bricks. Averagemeasurement 705 is determined using the difference between the pixelscorresponding to top boundary 703 and bottom boundary 704. The averagemeasurement between top boundary 703 and bottom boundary 704 arecompared to the known architectural standards for bricks dimensions (seeTable 1) to determine actual dimensions. In another embodiment, the leftboundary 706 and right boundary 707 of the brick are used to identifyerror and rescale and reconstruct the building model. Averagemeasurement 708 is determined using the difference between the pixelscorresponding to left boundary 706 and right boundary 707 and comparedto the known architectural standards for brick dimensions. For greateraccuracy of multiple smaller dimensioned architectural features (e.g.,bricks), averaging of an error associated with a large number of thebricks will increase accuracy. As with the door example, simple ratiosof width-to-height and height-to-width of the bricks are used (without apixel analysis) to match to known architectural ratios of standard bricksizes (with or without a threshold error).

FIGS. 8A, 8B, 8C, and 8D, collectively illustrate example embodimentsfor identifying error in scaling of a building model using roofingelements as the known architectural dimension in accordance with thepresent disclosure. For example, shingles, gables, chimneys, vent pipes,dormers, skylights, solar panels, fans, gutters, fascia boards, rakes,etc. are used as known roofing elements (known architectural features).

FIG. 8A illustrates an end view of a house with gable 801, chimney 802and vent pipe 803. The boundaries of the known architectural featuresare determined and a dimension (ratio) is extrapolated based on H:W(height/width) or W:H (width/height). In alternate embodiments, angles,such as θ₁ or θ₂, can be used to calculate dimensional ratios usingknown geometric analysis techniques. In addition, multiple dimensionalratios height:width:length (H:W:L) or known dimensions such as, depth,radius, diameter, circumference, perimeter, surface area, area, etc.,can be used without departing from the scope of the technology describedherein. For example, the ratio between the gable height and width isused to identify scale and determine the actual dimensions of the gableusing techniques in any of the various embodiments described herein. Aswith the previous examples, the measurements are compared to existingknown gable dimensional ratios, with or without threshold error rates,to identify possible known architectural standards matches.

FIG. 8A is used for diagrammatic purposes, specific positioning anddesignation of dimensions may change from image-to-image and distinctarchitectural feature. For example, the known chimney architecturalstandard may include or not include an upper collar or chimney cap. Foranother example, when a gable is analyzed, in one example embodiment,ratio or length measurements will only be taken for one side of thegable, and because of known symmetry, be considered applicable forscaling an opposing side or positioning a supporting wall. Also, thegable can be analyzed with or without an overhang, fascia, rake, etc.

FIGS. 8B, 8C and 8D each illustrate various shingle patterns. Theboundaries of the known architectural features are determined and adimension (ratio) is extrapolated based on H:W (height/width) or W:H(width/height). In an alternate embodiment, shingle patterns can includemultiple dimensional ratio analyses each covering identified uniqueshingle shapes within the shingle pattern. As with the previousexamples, the ratios/measurements are compared to existing known shingledimensional ratios, with or without scale error rates, to identify apossible known architectural standards match using techniques in any ofthe various embodiments described herein. FIGS. 8B, 8C and 8D are usedfor diagrammatic purposes, specific positioning, shapes, patterns,material types and designation of dimensions may change fromimage-to-image and with distinct architectural features.

FIG. 9 illustrates yet another example embodiment for identifyingpositions, scale, or scale error in a multi-dimensional building modelusing identified relative positions of elements as the knownarchitectural dimension. As previously described, if positions of keyknown architectural features within a multi-dimensional model can beidentified, the model can be accurately dimensioned (scaled) by properlymoving selected planes (e.g., walls) to correct positions, scaling orrescaling. For example, downspouts always follow an exterior wall edge.If the downspouts can be properly identified, proper placement ofassociated wall planes can be improved.

As shown, in an initial multi-dimensional model build, exterior wall 901is shown at a position 18 inches inset from roof overhang outer edge902. However, in an enlarged image of this section (903), it is shownthat an inside edge, plus gap between gutter and exterior wall, ofdownspout 904 is actually positioned at 18.8 inches from the overhangedge. The positioning of the downspout would suggest that the originalexterior wall placement was too close to the overhang edge. In a firstexample embodiment, without consideration of measurements, the exteriorwall plane in the model is simply moved to be adjacent to downspout 904.In a second embodiment, the scale error between an accurate positionmeasurement of 18.8 inches and the model placement of 18 inches can beused to scale/rescale the entire model. In a third embodiment, thisscale error can be applied simply to the associated plane orarchitectural features in the plane in which the downspout wasidentified.

FIG. 10 illustrates an embodiment of a flowchart for improving theaccuracy of the dimensions of a building model in accordance with thepresent disclosure. In one embodiment, the process of identifying scale,scale error, scaling, rescaling and reconstructing is used to improvethe accuracy of dimensioning a multi-dimensional building model. Process1000 includes identification of scale errors in a multi-dimensionalbuilding model by retrieving a digital image in step 1001 that containsarchitectural elements (i.e., siding, brick, door, window, gables,roofing features, etc.). In step 1002, architectural elements areidentified (e.g., doors). In step 1003, boundaries of the identifiedarchitectural element are defined and are used to measure the variousdimensions/dimensional ratios of the architectural elements in step1004. The measurement is compared to the known architectural elementstandard dimensions/dimensional ratios to determine the actualmeasurement in step 1005. Steps 1002 through 1005 are repeatable 1009 inan identification and measurement cycle 1006 that can be mathematicallycombined (e.g., averaged, mean, median, etc.) over multiple similar ordissimilar architectural features.

While siding, bricks, doors and windows can provide accurate scalingreferences, other objects located within the building model as well asin the foreground and background of digital images can be used toprovide increased scaling accuracy. For example, some additional objectsinclude, but are not limited to, parked automobiles, light fixtures,distance between door knob and base of doors, lamp posts, telephonepoles, stop signs, play structures, tables, chairs, or lawn equipment.Specific roof features include, but are not limited to, shingles,gables, eaves, dormers, gutters, chimney structures, pipe stacks,turbine vents and skylights. In other words, any known object (withidentifiable dimensions) in the foreground or background can count as adimension reference.

In one embodiment, the repeated comparison between the measurement ofmultiple selections of an architectural element (e.g., sidingmeasurements in various locations within the image) and the knownarchitectural standard dimensions established in step 1005 is fed into aweighted decision engine in step 1007 to determine an average scalingerror. The weighted decision engine in step 1007 uses learnedstatistical analysis to improve scaling over time and measurements. Asmore statistical information is accumulated (learned), the weighteddecision engine creates a more predictable result. In step 1008, thebuilding model is rescaled and reconstructed according to the decisiondetermined by the weighted decision engine in step 1007. Alternately,deep learning systems can be substituted for the weighted decisionengine without departing from the scope of the technology describedherein.

FIG. 11 illustrates an embodiment of a flowchart for weighing variousknown architectural elements to adjust scale in the dimensions of abuilding model in accordance with the present disclosure. Process 1100includes identification of error in a multi-dimensional building modelby retrieving a digital image in step 1101 that contains architecturalelements. In one embodiment, the repeated identification and measurementcycle 1102 is performed (1102(1), 1102(2) . . . 1102(n)) for multiplearchitectural elements (e.g., siding (1), brick (2), door (3), etc.)identified in the digital image retrieved in step 1101. For example, ifthe retrieved digital image includes more than one architectural element(e.g., a brick façade also showing the front door), identification andmeasurement cycle 1106 (repeated for averaging) is performed for eacharchitectural feature to determine which architectural feature(s) inweighted decision engine in step 1107 would likely provide the mostaccurate, rescaled building model in step 1108. For example, measuringthe front door may statistically prove a better gauge of scale error(e.g., closer to actual known measurements or having a higher frequencyof correlation to known dimensions over multiple cycles) than sidingscale determinations.

In one embodiment, a weighted decision engine is provided to determinethe architectural elements(s) that are most likely to produce anaccurately scaled/rescaled and constructed/reconstructed (1104) buildingmodel based on using a least quantity of processing or providing fastestcycle times, or that prove more accurate over time. In addition,location of architectural elements may determine specific façadescaling. For example, if a door on a façade indicates an error (4% toolarge) and bricks on a side façade indicate an error in width (3% toonarrow), the different facades could be rescaled separately.

In addition, the various embodiments may be interchangeably implementedbefore, during or after construction of the multi-dimensional model.

Referring now to FIG. 12 , therein is shown a diagrammaticrepresentation of a machine in the example form of a computer system1200 within which a set of instructions, for causing the machine toperform any one or more of the methodologies or modules discussedherein, may be executed. Computer system 1200 includes a processor,memory, non-volatile memory, and an interface device. Various commoncomponents (e.g., cache memory) are omitted for illustrative simplicity.The computer system 1200 is intended to illustrate a hardware device onwhich any of the components depicted in the example of FIG. 1 (and anyother components described in this specification) can be implemented.The computer system 1200 can be of any applicable known or convenienttype. The components of the computer system 1200 can be coupled togethervia a bus or through some other known or convenient device.

This disclosure contemplates the computer system 1200 taking anysuitable physical form. As example and not by way of limitation,computer system 1200 may be an embedded computer system, asystem-on-chip (SOC), a single-board computer system (SBC) (such as, forexample, a computer-on-module (COM) or system-on-module (SOM)), adesktop computer system, a laptop, notebook or tablet computer system,an interactive kiosk, a mainframe, a mesh of computer systems, a mobiletelephone, a smartphone, a personal digital assistant (PDA), a server,or a combination of two or more of these. Where appropriate, computersystem 1200 may include one or more computer systems 1200; be unitary ordistributed; span multiple locations; span multiple machines; or residein a cloud, which may include one or more cloud components in one ormore networks. Where appropriate, one or more computer systems 1200 mayperform without substantial spatial or temporal limitation one or moresteps of one or more methods described or illustrated herein. As anexample, and not by way of limitation, one or more computer systems 1200may perform in real time or in batch mode one or more steps of one ormore methods described or illustrated herein. One or more computersystems 1200 may perform at different times or at different locationsone or more steps of one or more methods described or illustratedherein, where appropriate.

The processor may be, for example, a conventional microprocessor such asan Intel Pentium microprocessor or Motorola power PC microprocessor. Oneof skill in the relevant art will recognize that the terms“machine-readable (storage) medium” or “computer-readable (storage)medium” include any type of device that is accessible by the processor.

The memory is coupled to the processor by, for example, a bus. Thememory can include, by way of example but not limitation, random accessmemory (RAM), such as dynamic RAM (DRAM) and static RAM (SRAM). Thememory can be local, remote, or distributed.

The bus also couples the processor to the non-volatile memory and driveunit. The non-volatile memory is often a magnetic floppy or hard disk, amagnetic-optical disk, an optical disk, a read-only memory (ROM), suchas a CD-ROM, EPROM, or EEPROM, a magnetic or optical card, or anotherform of storage for large amounts of data. Some of this data is oftenwritten, by a direct memory access process, into memory during executionof software in the computer 1200. The non-volatile storage can be local,remote, or distributed. The non-volatile memory is optional becausesystems can be created with all applicable data available in memory. Atypical computer system will usually include at least a processor,memory, and a device (e.g., a bus) coupling the memory to the processor.

Software is typically stored in the non-volatile memory and/or the driveunit. Indeed, for large programs, it may not even be possible to storethe entire program in the memory. Nevertheless, it should be understoodthat for software to run, if necessary, it is moved to a computerreadable location appropriate for processing, and for illustrativepurposes, that location is referred to as the memory in this paper. Evenwhen software is moved to the memory for execution, the processor willtypically make use of hardware registers to store values associated withthe software, and local cache that, ideally, serves to speed upexecution. As used herein, a software program is assumed to be stored atany known or convenient location (from non-volatile storage to hardwareregisters) when the software program is referred to as “implemented in acomputer-readable medium.” A processor is considered to be “configuredto execute a program” when at least one value associated with theprogram is stored in a register readable by the processor.

The bus also couples the processor to the network interface device. Theinterface can include one or more of a modem or network interface. Itwill be appreciated that a modem or network interface can be consideredto be part of the computer system 1200. The interface can include ananalog modem, isdn modem, cable modem, token ring interface, satellitetransmission interface (e.g., “direct PC”), or other interfaces forcoupling a computer system to other computer systems. The interface caninclude one or more input and/or output devices. The I/O devices caninclude, by way of example but not limitation, a keyboard, a mouse orother pointing device, disk drives, printers, a scanner, and other inputand/or output devices, including a display device. The display devicecan include, by way of example but not limitation, a cathode ray tube(CRT), liquid crystal display (LCD), or some other applicable known orconvenient display device. For simplicity, it is assumed thatcontrollers of any devices not depicted reside in the interface.

In operation, the computer system 1200 can be controlled by operatingsystem software that includes a file management system, such as a diskoperating system. One example of operating system software withassociated file management system software is the family of operatingsystems known as Windows® from Microsoft Corporation of Redmond, Wash.,and their associated file management systems. Another example ofoperating system software with its associated file management systemsoftware is the Linux™ operating system and its associated filemanagement system. The file management system is typically stored in thenon-volatile memory and/or drive unit and causes the processor toexecute the various acts required by the operating system to input andoutput data and to store data in the memory, including storing files onthe non-volatile memory and/or drive unit.

The technology as described herein may have also been described, atleast in part, in terms of one or more embodiments. An embodiment of thetechnology as described herein is used herein to illustrate an aspectthereof, a feature thereof, a concept thereof, and/or an examplethereof. A physical embodiment of an apparatus, an article ofmanufacture, a machine, and/or of a process that embodies the technologydescribed herein may include one or more of the aspects, features,concepts, examples, etc. described with reference to one or more of theembodiments discussed herein. Further, from figure to figure, theembodiments may incorporate the same or similarly named functions,steps, modules, etc. that may use the same or different referencenumbers and, as such, the functions, steps, modules, etc. may be thesame or similar functions, steps, modules, etc. or different ones.

While particular combinations of various functions and features of thetechnology as described herein have been expressly described herein,other combinations of these features and functions are likewisepossible. For example, the steps may be completed in varied sequences tocomplete the textured facades. The technology as described herein is notlimited by the particular examples disclosed herein and expresslyincorporates these other combinations.

The invention claimed is:
 1. A method of adjusting a three-dimensionalbuilding model, the method comprising: identifying a first architecturalelement associated with at least a first plane of a plurality of planesof a three-dimensional building model; identifying a position of thefirst architectural element coplanar with the first plane; determining ascale of the three-dimensional building model based on the identifiedfirst architectural element; determining a translation positional erroralong a normal of the first architectural element based on theidentified position of the first architectural element; moving the firstplane in a translation direction normal to the first architecturalelement based on the determined scale and the determined translationpositional error; and reconstructing the three-dimensional buildingmodel based on the moved first plane.
 2. The method of claim 1, furthercomprising scaling the reconstructed three-dimensional building modelbased on the determined scale.
 3. The method of claim 1, furthercomprising determining a scale error based on the moved first plane. 4.The method of claim 3, further comprising scaling the reconstructedthree-dimensional building model based on the determined scale error. 5.The method of claim 1, wherein there is a predetermined knownrelationship between the first architectural element and the firstplane.
 6. The method of claim 1, wherein determining the translationpositional error is further based on a predetermined known relationshipbetween the first architectural element and the first plane.
 7. Themethod of claim 1, wherein moving the first plane is further based on apredetermined known relationship between the first architectural elementand the first plane.
 8. One or more non-transitory computer-readablemedia storing instructions that, when executed by a system comprisingone or more processors, cause the one or more processors to performoperations comprising: identifying a first architectural elementassociated with at least a first plane of a plurality of planes of athree-dimensional building model; identifying a position of the firstarchitectural element coplanar with the first plane; determining a scaleof the three-dimensional building model based on the identified firstarchitectural element; determining a translation positional error alonga normal of the first architectural element based on the identifiedposition of the first architectural element; moving the first plane in atranslation direction normal to the first architectural element based onthe determined scale and the determined translation positional error;and reconstructing the three-dimensional building model based on themoved first plane.
 9. The one or more non-transitory computer-readablemedia of claim 8, further comprising scaling the reconstructedthree-dimensional building model based on the determined scale.
 10. Theone or more non-transitory computer-readable media of claim 8, furthercomprising determining a scale error based on the moved first plane. 11.The one or more non-transitory computer-readable media of claim 10,further comprising scaling the reconstructed three-dimensional buildingmodel based on the determined scale error.
 12. The one or morenon-transitory computer-readable media of claim 8, wherein there is apredetermined known relationship between the first architectural elementand the first plane.
 13. The one or more non-transitorycomputer-readable media of claim 8, wherein determining the translationpositional error is further based on a predetermined known relationshipbetween the first architectural element and the first plane.
 14. The oneor more non-transitory computer-readable media of claim 8, whereinmoving the first plane is further based on a predetermined knownrelationship between the first architectural element and the firstplane.